Three-axis-magnetometers (TAMs) are widely utilized as a key component of attitude determination subsystems and as such are considered the corner stone of navigation for low Earth orbiting (LEO) space systems. Precise geomagnetic-based navigation demands accurate calibration of the magnetometers. In this regard, a complete online calibration process of TAM is developed in the current research that considers the combined effects of environmental and instrumental errors including biases, non-orthogonally parameters, and the scale factors, without the need for clean room facilities. The sensor characteristics are estimated utilizing Kalman filter for a micro electro-mechanical sensor(MEMS)-based TAM standing on the experimental measured outputs in a noisy laboratory environment. Moreover, the stochastic TAM behavior is identified using the method of Allan variance analysis (AVA) through a six-hour static test. Subsequently, the nonlinear/non-Gaussian problem of attitude estimation, using a set of calibrated strap-down magnetometers is addressed utilizing the unscented particle filter (UPF), developed for the removal of colored-noise. Comparison of the estimated attitude, represented by quaternion parameters, with the true orientations demonstrates an acceptable level of accuracy of the developed calibration technique for small LEO space systems. Analysis of the root mean square error of the estimated attitude illustrates an accuracy of less than one degree for all axes. This is an ideal result, given the fact that MEMS-based magnetometers have been utilized.